Feedback driven adaptive combinatorial testing

The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects – failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them

Moving forward with combinatorial interaction testing

Combinatorial interaction testing (CIT) is an efficient and effective method of detecting failures that are caused by the interactions of various system input parameters. In this paper, we discuss CIT, point out some of the difficulties of applying it in practice, and highlight some recent advances that have improved CIT’s applicability to modern systems. We also provide a roadmap for future research and directions; one that we hope will lead to new CIT research and to higher quality testing of industrial systems

Keeping Secrets: Multi-objective Genetic Improvement for Detecting and Reducing Information Leakage

Information leaks in software can unintentionally reveal private data, yet they are hard to detect and fix. Although several methods have been proposed to detect leakage, such as static verificationbased approaches, they require specialist knowledge, and are timeconsuming. Recently, HyperGI introduced a dynamic, hypertestbased approach that detects and produces potential fixes for information leakage. Its fitness function tries to balance information leakage and program correctness, but as the authors of that work point out, there may be a tradeoff between keeping program semantics and reducing information leakage. In this work we ask if it is possible to automatically detect and repair information leakage in more realistic programs without requiring specialist knowledge. Our approach, called LeakReducer explicitly encodes the tradeoff between program correctness and information leakage as a multi-objective optimisation problem. We apply LeakReducer to a set of leaky programs including the well known Heartbleed bug. It is comparable with HyperGI on their toy applications. In addition, we demonstrate it can find and reduce leakage in real applications and we see diverse solutions on our Pareto front. Upon investigation we find that having a Pareto front helps with some types of information leakage, but not all

Modeling and Testing a Family of Surgical Robots: An Experience Report

Safety-critical applications often use dependability cases to validate that specified properties are invariant, or to demonstrate a counter example showing how that property might be violated. However, most dependability cases are written with a single product in mind. At the same time, software product lines (families of related software products) have been studied with the goal of modeling variability and commonality, and building family based techniques for both analysis and testing. However, there has been little work on building an end to end dependability case for a software product line (where a property is modeled, a counter example is found and then validated as a true positive via testing), and none that we know of in an emerging safety-critical domain, that of robotic surgery. In this paper, we study a family of surgical robots, that combine hardware and software, and are highly configurable, representing over 1300 unique robots. At the same time, they are considered safety-critical and should have associated dependability cases. We perform a case study to understand how we can bring together lightweight formal analysis, feature modeling, and testing to provide an end to end pipeline to find potential violations of important safety properties. In the process, we learned that there are some interesting and open challenges for the research community, which if solved will lead towards more dependable safety-critical cyber-physical systems

Covering array sampling of input event sequences for automated gui testing

This paper describes a new automated technique to generate test cases for GUIs by using covering arrays (CAs). The key motivation is to generate long GUI event sequences that are systematically sampled at a particular coverage strength. CAs, to date, have not been effectively used in sampling event driven systems such as GUIs which maintain state. We leverage a “stateless ” abstraction of GUIs that allows us to use CAs. Once the CAs have been generated, we reuse the abstractions to reinsert ordering relationships between GUI events, thereby creating executable test cases. A feasi-bility study on a well-studied GUI-based application shows that the new technique is able to detect a large number of previously undetected faults

Configurations everywhere: implications for testing and debugging in practice

us.abb.com Many industrial systems are highly-configurable, complicat-ing the testing and debugging process. While researchers have developed techniques to statically extract, quantify and manipulate the valid system configurations, we conjecture that many of these techniques will fail in practice. In this paper we analyze a highly-configurable industrial applica-tion and two open source applications in order to quantify the true challenges that configurability creates for software testing and debugging. We find that (1) all three appli-cations consist of multiple programming languages, hence static analyses need to cross programming language barriers to work, (2) there are many access points and methods to modify configurations, implying that practitioners need con-figuration traceability and should gather and merge meta-data from more than one source and (3) the configuration state of an application on failure cannot be reliably deter-mined by reading persistent data; a runtime memory dump or other heuristics must be used for accurate debugging. We conclude with a roadmap and lessons learned to help prac-titioners better handle configurability now, and that may lead to new configuration-aware testing and debugging tech-niques in the future

Metabolic Feedback Inhibition Influences Metabolite Secretion by the Human Gut Symbiont Bacteroides thetaiotaomicron

Microbial metabolism and trophic interactions between microbes give rise to complex multispecies communities in microbe-host systems. Bacteroides thetaiotaomicron (B. theta) is a human gut symbiont thought to play an important role in maintaining host health. Untargeted nuclear magnetic resonance metabolomics revealed B. theta secretes specific organic acids and amino acids in defined minimal medium. Physiological concentrations of acetate and formate found in the human intestinal tract were shown to cause dose-dependent changes in secretion of metabolites known to play roles in host nutrition and pathogenesis. While secretion fluxes varied, biomass yield was unchanged, suggesting feedback inhibition does not affect metabolic bioenergetics but instead redirects carbon and energy to CO2 and H2. Flux balance analysis modeling showed increased flux through CO2-producing reactions under glucose-limiting growth conditions. The metabolic dynamics observed for B. theta, a keystone symbiont organism, underscores the need for metabolic modeling to complement genomic predictions of microbial metabolism to infer mechanisms of microbemicrobe and microbe-host interactions

End-to-end Molecular Communication Channels in Cell Metabolism: an Information Theoretic Study

The opportunity to control and fine-tune the behavior of biological cells is a fascinating possibility for many diverse disciplines, ranging from medicine and ecology, to chemical industry and space exploration. While synthetic biology is providing novel tools to reprogram cell behavior from their genetic code, many challenges need to be solved before it can become a true engineering discipline, such as reliability, safety assurance, reproducibility and stability. This paper aims to understand the limits in the controllability of the behavior of a natural (non-engineered) biological cell. In particular, the focus is on cell metabolism, and its natural regulation mechanisms, and their ability to react and change according to the chemical characteristics of the external environment. To understand the aforementioned limits of this ability, molecular communication is used to abstract biological cells into a series of channels that propagate information on the chemical composition of the extracellular environment to the cell’s behavior in terms of uptake and consumption of chemical compounds, and growth rate. This provides an information-theoretic framework to analyze the upper bound limit to the capacity of these channels to propagate information, which is based on a well-known and computationally efficient metabolic simulation technique. A numerical study is performed on two human gut microbes, where the upper bound is estimated for different environmental compounds, showing there is a potential for future practical applications

Metabolic Synergy between Human Symbionts \u3ci\u3eBacteroides\u3c/i\u3e and \u3ci\u3eMethanobrevibacter\u3c/i\u3e

ABSTRACT Trophic interactions between microbes are postulated to determine whether a host microbiome is healthy or causes predisposition to disease. Two abundant taxa, the Gram-negative heterotrophic bacterium Bacteroides thetaiotaomicron and the methanogenic archaeon Methanobrevibacter smithii, are proposed to have a synergistic metabolic relationship. Both organisms play vital roles in human gut health; B. thetaiotaomicron assists the host by fermenting dietary polysaccharides, whereas M. smithii consumes end-stage fermentation products and is hypothesized to relieve feedback inhibition of upstream microbes such as B. thetaiotaomicron. To study their metabolic interactions, we defined and optimized a coculture system and used software testing techniques to analyze growth under a range of conditions representing the nutrient environment of the host. We verify that B. thetaiotaomicron fermentation products are sufficient for M. smithii growth and that accumulation of fermentation products alters secretion of metabolites by B. thetaiotaomicron to benefit M. smithii. Studies suggest that B. thetaiotaomicron metabolic efficiency is greater in the absence of fermentation products or in the presence of M. smithii. Under certain conditions, B. thetaiotaomicron and M. smithii form interspecies granules consistent with behavior observed for syntrophic partnerships between microbes in soil or sediment enrichments and anaerobic digesters. Furthermore, when vitamin B12, hematin, and hydrogen gas are abundant, coculture growth is greater than the sum of growth observed for monocultures, suggesting that both organisms benefit from a synergistic mutual metabolic relationship. IMPORTANCE The human gut functions through a complex system of interactions between the host human tissue and the microbes which inhabit it. These diverse interactions are difficult to model or examine under controlled laboratory conditions. We studied the interactions between two dominant human gut microbes, B. thetaiotaomicron and M. smithii, using a seven-component culturing approach that allows the systematic examination of the metabolic complexity of this binary microbial system. By combining high-throughput methods with machine learning techniques, we were able to investigate the interactions between two dominant genera of the gut microbiome in a wide variety of environmental conditions. Our approach can be broadly applied to studying microbial interactions and may be extended to evaluate and curate computational metabolic models. The software tools developed for this study are available as user-friendly tutorials in the Department of Energy KBase

Genetic Improvement @ ICSE 2020

Following Prof. Mark Harman of Facebook's keynote and formal presentations (which are recorded in the proceedings) there was a wide ranging discussion at the eighth international Genetic Improvement workshop, GI-2020 @ ICSE (held as part of the International Conference on Software En- gineering on Friday 3rd July 2020). Topics included industry take up, human factors, explainabiloity (explainability, jus- tifyability, exploitability) and GI benchmarks. We also con- trast various recent online approaches (e.g. SBST 2020) to holding virtual computer science conferences and workshops via the WWW on the Internet without face to face interac- tion. Finally we speculate on how the Coronavirus Covid-19 Pandemic will a ect research next year and into the future